DOCUMENT RESUME 



ED 301 596 



TM 012 583 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Eason, Sandra; Thompson, Bruce 

The Effect of Persuasion on the Perception of Program 
Evaluation Information. 
Oct 88 

36p.; Paper presented at the Annual Meeting of the 
American Evaluation Association (New Orleans, LA, 
Octot»er 26-^29, 1988). 
Reports - Research/TechniCctl (143) — 
Speeches/Conference Papers (150) 

MF01/PC02 Plus Postage. 
* Administrator Attitudes; Administrators; 
♦Credibility; *Educationca Assessment; *EVcauation 
Problems; Evaluation Utilization; Svaauators; 
Persuasive Discourse; *Progi:am Evc^uatibn 
*Evcauatibn Reports 



ABSTRACT 

The vicUDility of persuasion theory as a framework for 
the assessment of program evc^.uation utilization was assessed. 
Independent variables studied included credibility, involveinent, and 
role groups. More specif icrlly, the research was designed to address 
three questions concerning: (1) the occurrence of peripheral 
involvement persuasive effects regardless of the impact of the 
evaluation report on itis readers; (2) the Oucurrence of evaluator 
peripheral credibility cue effects independently of involvement 
effects; and (3) the differenticQ. sensitivity of high-level 
administrators and licensed evaluators to peripheral cue information. 
Subjects included 20 administrators working with loccG. educationetl 
agencies in Louisiana and 51 program evcduators. The subjects' 
responses to simulated evc^.uation reports and quest ionnciires were 
assessed. Results indicate that evcduat or credibility is perceived by 
report readers and that these perceptions may even affect 
counterargumentation. Furthermore, counterargument may lead to 
inaction or delay even when dealing with favorably received 
recommendations* The findings also indicate that issue-involvement 
intervention did not directly affect the perception of involvement; 
attitude change is a function of the extent and direction of 
cognitive responses; and program evaluators and high-level 
administrators are more similar than dissimilar in their perceptions 
of evaluation information. (TJH) 
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THE EFFECT OF PERSUASION ON THE PERCEPTION 
OF .PROGRAM EVALUATION INFORMATION 
As the importance of program evaluation has increased, 
problems related to utilization of the evaluation reports have 
been more widely recognized. The traditional view during the 
1960 *s was that program evaluations provided information for 
action (Weiss, 1972). Evaiuators expected that their reports 
would lead to major program change when problems were detected. 
Concern grew, however, because evaiuators found that their 
efforts resulted in little direct utilization (Alkin, Daillak, 
& White, 1979, pp. 14-16; Thompson & king, 1981, pp. 3-4). As 
a result, program evaluation theorists began to question 
whether these action expectancies were realistic. They also 
sought to determine if the use and non-use of evaluations by 
decision makers had discernible patterns i What emerged was a 
new and expanded definition of utilization and of how 
utilization related to the complexities of organizational 
decisions. 

One of the first to acknowledge a problem with th^ action 
concept was Patton et al. (1977), who noted that "evaluation 
literature has focused on too nairdw a definition of evaluation 
research impacts" (p. 161). The perception of underuti lization 
ultimately led evaluation theorists to redefine utilization 
(Brown, Newman, & Rivers, 1984; Leviton & Hughes, 1981). Two 
major distinctive definitions for program evaluation 
utilization surfaced. Program utilization is viewed from both 



a narrow and a broad perspective. If evaluation information is 
meant for immediate and direct use in improving the quality of 
a program, then use of the narrow perspective is indicated. 
However, when evaluation findings are used as supplemental 
information for future decisions, the broader perspective of 
use is more appropriate. Additional research from the late 
1970 's also indicates that the broader perspective is becoming 
more commonly used by evalators (Alkin, Daillak, & White, 1979; 
Knorr, 1977; Patton, 1978). With the expanding definitions, 
utilization becomes "less dramatic and more difficult to 
explicitly Treasure and demonstrate. It represents a view of 
evaluation in which the role of human interaction in the 
communication process is given more credance" (Brown & 
Braskamp, 1980, p. viii). 

Because of the concern for program evaluation utilization, 
researchers (Leviton & Hughes, 1981, p. 528) have 
systematically differentiated several catagories of 
utilization: instrumental, conceptual, and persuasive. 
Instrumental use is an ideal. In instrumental use an 
evaluation is completed and direct discernible changes occur. 
Conceptual use of research information was defined by Rich 
(1977) as "influencing a policymaker's thinking about an issue 
without putting information to any specific, documentable use" 
(p. 200). 

Persuasive use "draws on evaluation evidence in attempts 
to convince others to support a political position, or to 
defend such a position from attack" (Levi on & Hughes, 1981, p. 
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528). Buc as Patton (1978) states, "the traditional academic 
values of many social scientists lead them to want to be 
nonpolitical in their research. Yet they always want to affect 
government decisions. The evidence is that they cannot have vt 
both ways" (p. 46) . 

Evaliiators have a responsibility to provide not only a 
comprehensive quality evaluation report but the additional 
responsibility of increasing the utilization of evaluation. So 
more evaluatprs have now come to appreciate House's (1977, p. 
5) poisition that one of the primary functions of evaluations is 
to persuade. For example, following a five-year series of 
studies, Newman, Brown and Braskamp (1980) argued that the 
"evaluation reporting process can be viewed as analogous to 'a 
persuasive communication message" (p. 29). Thompson (1981), in 
his review of communication theory and program evaluation, 
emphasized applying theoretically grounded persuasion 
principles to establish a possible relationship between 
persuasive effort and program evaluation utilization. 

The purpose of the present study was to investigate the 
effects of several variations on perceptionr of evaluation 
information. High-level school district administrators and 
licensed program evaluators were the participants in the study. 
The study was grounded in contemporary persuasion theory, and 
the experimental variations examined were selected on this 
basis. 

Contemporary Persuasion Theory 
Four primary theoretical approaches constitute 



contemporary persuasion theory. These include learning, 
consistency, perceptual, and functional approaches (Shelby, 
1986, p. 10). The first approach, learning theory^ attempts to 
explain or predict the relationship between a stimulus and a 
response (Staats, 1967, p. 373). The stimulus may be the 
source, the message, or the channel. The response is how the 
receiver reacts to the stimulus. 

Several persuasion models have developed from learning 
theory. Two of the most commonly used are the information 
processing and cognitive response models (Shelby, 1986, p^ 11). 
Information processing models developed by Hovland and 
associates received much attention in the 1950 's and 1960 's. A 
more recent theory^ the cognitive response model developed by 
Brock (1967) and Greenwald (1968) and refined by Petty (1977), 
focuses on the receiver's cognitive response to a message. The 
theory hypothesizes that thoughts generated by the initial 
persuasive communication influence the receiver's attitude 
concerning acceptance or rejection of the message. The 
persistence of attitude is a function of the elaboration 
created by the cognitive responses to the message. The 
principles of this theory provided one important framework for 
the present study. 

Consistency theory, a second approach to persuasion, 
focuses on the relationship between the stimulus and the 
receiver's frame of reference. This is in contrast to learning 
theory, which focuses on the stimulus and the response. How 
much a receiver has stored in memory about the message content 



and/or the source affects the perceived persuasive intent. 

Dissonance, a cpnstruct of consistency theory, has been 
applied to persuasion theory. Dissonance involves the 
inconsistency found to exist when new environmental phenomena 
confront previously formed, internally consistent knowledge, 
opinions, and attitudes of a person (Festinger, 1957). When 
dissonance occurs, tension or discomfort develops which will 
then motivate the person to achieve consistency or consonance. 
For example, cognitive dissonance occurs when a person who 
smokes is shown a graphic film of the health* perils of smoking 
To relieve tension, the person rationalizes that quitting 
smoking \eads to weight gain which is also hazardous to health 
Smoking is continued and the person has achieved consonance 
with the habit. 

The third major theoretical approach to persuasion is the 
perceptual approach. It emphasizes how receivers perceive the 
message. The perceptual approach focuses in particular upon 
the receiver's atcitude or frame of reference about the 
message. The approach emphasizes that it is the perception of 
the message rather than the actual message that affects 
attitude change. 

Fourth, the functional approach to persuasion states that 
potential persuasive effects must be relevant to the needs of 
the receiver (Katz, 1960). Katz hypothesized that attitudes 
serve at least one of the following functions: adjustment, 
ego-defense, value-expression, and knowledge acquisition (p. 
461). An important theory in the functional approach. 



compliance-gaining, assumes that power is necessary for 
persuasion to occur (Carlsmith, Collins & Helmrieich, 1966, p. 
333) • 

The Elaboration Likelihood Model 
Given the various theoretical approaches to persuasion and 
the diversity of data on tlie traditional source, message, 
receiver, and channel variables. Petty and Cacioppo (1986) 
developed a general theory of attitude change called the 
Elaboration Likelihood Model (ELM) • The ELM attempts to 
integrate the many conflicting findings and theories from . 
previous work into one viable framework, 

"Elaboration" is defined as the degree to which a person 
cognitively procf^sses the message, "Likelihood" refers to the 
person's ability and motivation. Therefore, if a person has 
the ability and is motivated to process issue-relevant 
information, elaboration likelihood is high. Conversely,, if 
ability and motivation are not present^ elaboration likelihood 
is low. 

The ELM divides the previous empirical findings and 
theories into two "routes to persuasion" (Petty & Cacioppo, 
1986, p, 3): central and peripheral. Within the central route 
a receiver is led by ability and motivation t * cognitively 
consider the issues relevant to the message. The theorists 
hypothesize that the effort involved in mentally processing 
information makes persuasion through the central route more . 
permanent and more resistant to change than the peripheral 
route, A central route is involved when a voter critically 

6 

I 

8 



analyzes the oolitical platform of a candidate rather than the 
candidate's projected charisma. 

Conversely, in the peripheral route a receiver is led to 
react to a simple cue, rather than by cognitively processing 
issues. The credibility of the source, the topic of the 
communication, the attractiveness of the source, the length of 
the message, and the media type are only a few types of cues. 
In the previous example, the peripheral route might involve 
voters watching a TV commercial and focusing on the candidate 
shown with a happy family. 

One of the basic differences between central and 
peripheral routes is the permanency of their effects on 
attitude change. Peripheral cues are easier to process, but 
because they lack cognitive press they are less resistant to 
competing messages. However, the power of cues can lead to 
persuasion even in the absence of issue-relevant information. 
For example, an automobile dealer may ignore the relative 
quality of a product but still persuade the buyer with the 
simple message, "Buy American." Conversely, central arguments 
are difficult to establish but do result in longer lasting 
attitudinal change. The ELM predicts that when both the source 
and message contain combinations of central issue-relevant 
arguments and peripheral cues, the power to persuadvS is 
increased. 

The ELM also indicates that the relative effectiveness of 

persuasion techniques is, in part, a function of the 

characteristics of the receiver. Receivers may be 
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differentiated by bias, motivation, prior knowledge, locus of 
control, and intelligence. Additionally, different receivers 
have different needs for information and different kinds of 
decisions to make (Newman, Bull, Brown, & Rivers, 1986). In 
the present study different receivers, high-level 
administrators and certified program evaluators, were the 
audience. 

I 

Literature on the comparison of administrator and 
evaluator views of evaluation is scarce. Thompson and Miller 
(1984) explored these two perceptions of evaluation by 
employing Meltsner's model. The research question was whether 
administrators and evaluators had similar views of evaluation 
and the evaluator 's .role* Two of. the Q-technique factors 
consisted of a mix of both groups. The results suggest that 
some administrators and evaluators hold similar views of 
evaluation* Also, since most administrators grouped in the 
first two subject clusters, the results suggest that 
administrators are more homogeneous in their views of program 
evaluations. The views of the evaluators were not as 
consistent. Presumably evaluators and administrators can 
communicate more adequately if they understand each other's 
views of program evaluation. The present study was conducted 
in part to compare the two groups* perceptions of evaluation 
information* 

Two persuasive elements were the primary focus in the 
present study. These were source credibility and issue 
involvement. With respect to source credibility, attribution 
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theory suggests that source characteristics have an impact on 
message acceptance or rejection* The source factor can affect 
persuasion in several different wayss serving as arguments, 
serving as cues, or affecting the extent op direction of issue 
and argument processing (Petty & Cacioppo, 1986, p. 47)* 
Under varying conditions of high or low elaboration 
likelihood, the source factor can affect attitude change* When 
a person is unmotivated and unable to evaluate message 
arguments, simple cues such as source credibility are relied 
upon. Thus, under low elaboration likelihood a highly credible 
source tends to enhance persuasion. But when an audience 
member is highly motivated and able to cognitively process the 
issue-relevant arguments, the effects of source credibility 
tend to disappear^ These principles have been corroborated in 
experimental research (Maddux & Rogers, 1980; Pallak & Francia, 
1985). 

A second variable, issue involvement, is a motivational 
variable affecting elaboration likelihood (Petty & Cacioppo, 
1986, p. 81). Issue involvement, also referred to as personal 
relevance, concerns the extent to which the topic of the 
message is of relevance to the receiver. High personal 
relevance of a policy change in school dress code occurs if the 
change takes place in a receiver's own school. Low personal 
relevance might be typified by a change taking place in another 
state. 

Involvement is based on social judgment theory which 
posits latitudes of acceptance, rejection, or noncommitment. 
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After reviewing the persuasion research. Petty and Cacioppo 
(1986, p, 82) found that few researchers had considered the 
effects of issue-relevant arguments on persuasion. For 
example, issue relevance on the topic of a new exam policy 
might be hi |h for some undergraduate students but the topic of 
import duties on raw silk miaht be less relevant to these 
students. 

If the information is consistent with the subject'^s 
initial opinion, processing of the strengths of the message 
will occur. However, if the information is inconsistent with 
the subject's opinion, high relevance topics will generate 
counterarguments to the message. Cons quently. Petty and 
Cacioppo (1986) incorporated personal relevance as a central 
principle to their ELM. As personal relevance increases, 
people become more motivated i » work harder to process 
issue-relevant arguments (Pett^ & Cacioppo, 1979a, p, 1915), 
The more involved the receiver is in the outcome, the more 
resi^jtant the receiver will also be to changing attitudes, 
Aga^n, previous research supports these theoretical principles 
(Morton, 1986? Petty & Cacioppo, 1979b), 

In summary, the literature portrays persuasion theory as 
a viable framework for investigation regarding more effective 
program evaluation utilization. Several studies have been 
reported involving persuasion *n studies of program evaluation 
utU^7ation, For example, a series of studies conducted by 
Brown and Braskamp a980) present <d evidence suggesting that 
persuasion th.^ory provides a viable framework for studying 
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program evaluation utilization. 

The present study, employed a 2 x 2 x 2 (credibility 
manipulation x involvement manipulation x role group) design to 
investigate the effects of the three independent variables on 
ascribed eyaliiator credibility and particip.^nt issue 
involvement* Specifically, the study was designed to address 
three research questions^ First, will peripheral involvement 
persuasive effects occur even if evaluation information does 
not directly and immediately impact evaluation report readers? 
Second, will evaliiator peripheral credibility cue effects occur 
independently of involvement effects? Third, will high-level 
adiuinistrators and licensed evalcators be differentially 
sensitive to peripheral cue information? A qualitative 
componant of the investigation focused on participant thought 
processes when reading the evaluation report* The thought 
processes of interest were both counterargument and cognitive 
processing of the central evaluation message* Cognitive 
response theory (Brock, 1967; Greenwald, 1968; Petty, 1977) 
suggests that cognitive press is an important aspect of 
persuasive effort. 

Method 

Subjects 

The subjects in the present study were either high-level 
administrators or program c^valuators. Administrators were all 
123 Louisiana local education agency personnel with 
••superintendent," "assistant superintendent," "associate 
superintendent," or "deputy superintendent" in their titles. 
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The 123 high-level administrators representee} the entire 
population of high-level administrators across the state of 
Louisiana. 

Louisiana is somewhat unique in that the state provides 
for tormal licensure of educational program evaluatprs. 
Program evaluators in the present study were selected from all 
certified program evaluators currently employed in Louisiana. 
Three hundred evaluators were randomly selected from the total 
state list of 443. 

Evaluation reports and questionnaires were mailed to a 
combined total of 423 subjects. Questionnaires for 11 
evaluators were returned as non-deliverable. Questionnaires 
were received from 20 of the 123 high-level administrators for 
a response rate of 16.3%. The response rate for the evaluators 
was similar with 51 out of 289 responding (17.6%). The total 
response rate (n = 71) for the combined groups was 17.2%. 

Response rates on these orders are not unusual in this 
sort of research (Kerlinger, 1986, p., 380). To an extent 
response rate was viewed somewhat as a cost of greater external 
validity. Previous empirical studies (Thompson & Levitov, 
1983) of evaluation use research indicate that too many 
researchers have employed samples v.^f convenience of university 
graduate students in their research. In the present study a 
concerted effort was made to gather data from a statewide pool 
of people who were actually high-level administrators or 
certified program evaluators. 

However, it is important to try to establish that the 
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final sample was reasonably representative of the population as 
a whole. In the present study geographic area was available as 
a variable that could be used as for these purposes, without 
compromising subject anonymity* 

The geographic distribution of the role groups is 
presented in Figure !• The 0* S* Postal Service divides the 
state into nine geographical areas by zip codes with each area 
designated by a large city* The sample represented all areas* 
Responses were received from seven of the nine areas* The map 
suggests that within the seven areas the rate of returns was 
roughly representative of the distribution of the role groups* 



INSERT FIGURE 1 ABOUT HERE 



Procedure 

A simulated report was utilized to investigate the 
persuasive effects of source credibility and issue involvement 
on message acceptance* In order to make the report 
believable, a nonprofit company was created* The company's 
name was printed on professional stationary* The simulated 
evaluation described a five-year "pilot" study of a career 
option program* The report described the program, objectives, 
and the evaluation results and recommendations* 
Hypothetically, the five-step career option plan had been 
implemented in seven school districts of a southern state* The 
report was identical for all subjects* Subjects were asked to 
.help evaluate the quality of the report before the report was 
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to be distributed to other educators. 

The experimental manipulation occurred in the cover 
letter, and involved the variables of source credibility and 
issue involvement. The inclusion of the intervention in the 
cover letter was to make the manipulation less obtrusive and to 
minimize reactivity. 

Our operational descriptions for the variables of source 
credibility and issuo involvement have been used in several 
previous studies (Maddux & Rogers, L980; Norton, 1986; Petty, 
Cacioppo, & Heesacker, 1981). These descriptions were adapted 
for the present study. The evaluator titles, "researcher" ani 
"art specialist," were determined from previous evaluation use 
research to have high and low evaluator credibility, 
respectively (Braskamp, Brown, & Newman, 1978). In the present 
study the evaluator described ^as an art specialist was defined 
as a specialist in the field of music. In the high credibility 
condition the cover letter described the evaluator as a 
professor of educational research. 

Issue involvement, also referred to as personal relevance, 
was manipulated in the cover letter as well. Both involvement 
conditions were designed to not involve direct immediate 
impacts on the subjects. The evaluation report presented to 
the more highly-involved subjects was preceded by information 
that the subject was to help evaluate the quality of the report 
before the report was to be distributed to educators, civic 
organizations, and government officials within Louisiana and 
other southern states. Thus, the possibility of future 
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involvement within the same geographic region was indicated. 

The message for the low involvement group was preceded by 
similar instructions. However, subjects in the low involvement 
condition were told that the results were to be distributed in 
Wisconsin, and that results of the evaluation would not be 
employed for some time. Thus, involvement was portrayed as 
being remote in both time and place. 

Previous empirical research has clearly established that 
stark contrasts of no involvement versus immediate, direct, 
personal involvement do produce discernable differences in 
message processing. But direct immediate involvement is not 
characteristic of real evaluation settings in which decision 
responsibility is often shared and in which movement toward 
decisions is often incremental. Our purpose in designing the 
involvement intervention was to produce a more ecologically 
valid intervention involving a more realistic contrast. Such 
research may also be useful in defining the limen at which 
involvement effects occur. 

An 11-point Likert scale was employed to assess the 
effectiveness of the manipulation of source credibility and 
issue involvement. The same scale has been used in previous 
research (Norton, 1986; Petty & Cacioppo, 1979; Pallak & 
Francia, 1985; Swasy & Munch, 1985). The questionnaire 
included 13 items to measure direct manipulation effects, 
divided into the following catagories: five items measuring 
credibility, four items to assess adequacy of involvement, and 
the remaining four items in this section acted as fillers used 
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to lessen reactivity* The credibility and involvement items 
were summed to £orm two overall composite scores that measured 
each, dependent variable; Four items in this section were 
reversed phrasfed to minimize response set* 

Other procedures similar to these utilized in studies by 
Petty and Cacioppo (1979ar p. 1918) and Norton (1986, p. 43) 
were also followeid^ thus, subjects were also asked to indicate 
their personal attitude about the topic, i.e., career option 
plans, their agreement with the evaluator^s recoromendations, 
and listed the thoughts they had while reading the report* 

Thought listing is a cognitive response and can be 
utilized as an important dependent variable (Cacioppo & Petty, 
1981), as suggested previously. Assessing cognitive thought 
responses is a form o£ attitude measurement. In the present 
study cognitive responses were defined as "those thoughts that 
pass through a person's mind as he or she anticipates, 
receives, or reflects upon a message designed to change 
beliefs, attitudes, or behaviors" (Cacioppo & Petty, 1981, p. 
310). The technique, developed by Brock (1967) and Greenwald 
(1968), has received considerable empirical support (Chaiken, 
1980; Petty, Harkins, & Williams, 1980; Swasy & Munch, 1985). 
In thought listing, the subject attempts to relate new 
information to existing knowledge. Positive, negative, or 
neutral thoughts about the target issue may be generated, to 
the degree that the person has positive cognitive responses 
(favorable thoughts), the more the person will agree with the 
communication. However, if the person generates more negative 
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cognitive responses (unfavorable thoughts) , then the person may 
disagree with the communication. 

Results 

Preliminary analyses were performed to test the 
reliability of each of four scales. The scales were: 
credibility, with five items; involvement, four items; 
attitude, four items; and five items measuring reaction to 
report recommendations. The alpha reliability coefficients 
were .80 for credibility, .93 for involvement, and .88 for 
attitude. These reliability coefficients are sufficiently high 
to allow the detection of systematic response variance and thus 
statistical significance. However, the recommendation scale's 
alpha reliability cpefficient was .65. The result was 
considered ah artifact of the diverse content in the 
recommendation items, and so separate analyses were conducted 
across each of these five items. Descriptive statistics on the 
credibility and involvement conditions are presented in Table 
1. 
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The study's first research question asked whether 
peripheral involvement persuasive effects occur even if 
evaluation information does not directly and immediately impact 
evaluation report readers? "Involvement" was measured using 
four 11-point Likert-scale items (e.g., Norton, 1986; Swasy & 
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Munch, 1985) with the extreme descriptive anchors, "agree 
strongly" or "disagree strongly". The mean for the high 
invpiveraent condition was 21.78 (SD = 13.51) and the mean for 
the low involvement condition was 22*50 (SD « 14.44).* A 
univariate test of these differences was not statistically 
significant (F (1,63) « .05, £ >.05). This finding indicates 
that the subjects did not perceive any difference between the 
high and low involvement conditions* No other main or 
interaction effects were statistically significant (alpha « 
•05) for this dependent variable* Since the scale ranged from 
four (most involved) to 44 (least involved), and the means for 
the two involvement conditions were near the middle of this 
raige, both subject groups were mcdr^'rately involved in 
processing the evaluation report information. 

The study's second research question asked, do evaluator 
peripheral credibility cue effects occur independently of 
involvement effects? As noted previously, the subjects rated 
the extent to which they found the source "credible". The 
rating system utilized an 11-poirit scale where "1" indicated 
"agree strongly" and "11" indicated "disagree strongly*" Five 
statements were combined so that maximum acceptance of the 
source equalled five and least acceptance equalled 55. 
Subjects in the high credibility condition rated the evaluation 
report source as being more credible (M = 23.64, SD = 8.00) 
than subjects in the low credibility condition (M = 33.19, SD 
=9.49). This difference in means was statistically significant 
(F (1,63) « 21.40, £ <.001); the correlation ratio effect size 



18 

20 



for this analysis was 24.1% (1660.95/6893.12). in a 2 x 2 x. 2 

* 

factorral analysis of variance, no other univariate null 
hyipotheses involving effects on the credibility dependent 
variable were rejected. 

The study's third research question asked, are high-level 
administrators and licensed evaluators differentially sensitive 
to peripheral cue information? To address the third research 
question, and to address the first two research questions in a 
multivariate context (Fish, 1988), a 2 x 2 x 2 MANOVA involving 
the two dependent variables, credibility and involvement, was 
conducted. Only the null hypothesis involving the credibility 
main effect was rejected (lambda = .77^ df »2/62, £ <.001). 

Ancillary analyses were performed to test for other 
effects that were also of interest in the study. Individual 
analyses ware computed for each of the five recommendation 
items, rather than for a total recommendation score, since 
these scores were not very interrelated, as noted previously. 
The results indicated that the subjects were generally positive 
in their acceptance of the report recommendations. Thus, 
subjects were open to the topic in the evaluation report. 

Scores on the credibility and involvement scales were 
negligibly related (r = -.07, £ > .05). Scores on two of the 
recommendation items were related 'to perceptions of evaluator 
credibility (r = .30; r = .21). The multiple correlation (R » 
.36) between scores on the credibility scale and the five 
recommendation items was not statistically significant (F » 
1.97, df « 5/65, £ ^ .09). Scores on the involvement scale 
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we're significantly related to scores on two recommendation 
items, one (r « •40) also significantly associated with the 
involvement scale, and another recommendation item (r « .21) • 
The multiple correlation (R - .48) between scores on the 
involvement scale and the five recommendation items was 
statistically significant (F « 4i02, djE 5/65, £ < .05). 

The second series of ancillary analyses was performed on 
the attitude scale. Since the four items produced reasonably 
internally consistency scores (alpha reliability « .88), these 
analyses employed summated scale scores. The three-way 
factorial janalysis of variance on attitude toward career option 
plans involved no statistically significant main effects or 
interactions. The findings indicated that the attitudes toward 
career option plans were similar across desirin conditions. The 
grand mean (13.56? SD « 7.64) indicated a moderately positive 
attitude toward career option plans, since these items were 
rated on a 9-point scale (lower scores being more favorable) 
such that total scale scores therefore ranged from four to 36. 

The third ancillary analysis involved a primary purpose of 
the study. This analysis focused on the data from the 
thought-listing procedure, and was grounded in the cognitive 
response model (Petty, 1977), as noted previously. According 
to the Elaboration Likelihood Model (ELM), persuasion is 
affected by the extent that subjects cognitively process 
issue-relevant information contained in a messagfe. Thought 
listing is the recommended procedure used to assess how much 
message' processing the subjects conducted (e.g., Cacioppo & 
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Petty, 1981) • Data were collected on the number of favorable^ 
unfavorable, and neutral or irrelevant thoughts generated by 
the subjects toward the evaluation* Our procedures were again 
modelled on those employed in previous research. 

To obtain an indication of the f avorableness of the 
cognitions, cognitive response ratings were summed. For 
example, two positive thoughts, three negative thoughts, and 
two neutral thoughts would sum to -I, An analysis of variance 
on the cognitive response data yielded a statistically 
significant main effect. The main effect for the credibility 
manipulation indicated that role groups produced more 
counterarguments, i.e., negative arguments, when the source was 
perceived as low credible (M = -1.68) than when the source was 
perceive as highly credible (M » -0.41) (F (1,63) » 5.74, £ 
<.05) c 

A second cognitive response focused on the total number of 
issue-relevant thoughts generated in different experimental 
conditions. The number was obtained by simply adding the 
number of non-neutral issue-relevant thoughts generated by each 
subject. For example, two positive, three negative, and two 
neutral thoughts would add to five total non-neutral thoughts. 
There were less thoughts produced for a highly credible source 
(M » 1.9:7, SD * 1.46) than for a source with low credibility (M 
a 3.26, SD »2.38) (F (1,63) « 8.03, £ <.01). A low credible 
source generated more total counterarguments than a high 
credible source^ Additionally, a three-way interaction of 
credibility by involvement by role group yielded a 
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statistically significant effect (F (1,63) » 6.14/ £ <.02). 

Discussion 

Conclusions important to the utilization of program 
evaluations may be drawn from the present study- First, the 
study's research question concerning the detection of source 
credibility information was supported by the findings of this 
study. The findings were in agreement with those of Hovland 
and Weiss (1951) who observed that source credibility is 
attended to by subjects. Also, the source credibility findings 
were in agreement with those in the related program eval.uation 
research of Brown, Bfaskamp, and Newman (1978). 

These findings suggest the importance "of attehding to 
Standard A2 of the Joint Committee on Standards for Educational 
Evaluation (1981, pp. 24-26). This standard requires that "The 
persons conducting the evaluation should be both trustworthy 
and competent to perform the evaluation, so that their findings 
achieve maximum credibility and acceptance." The results of 
the present study, as reported in Table 1, suggest that both 
evalua^ors and high-level administrators do attend to 
information about evaluator experience and job title when 
making judgments about evaluator credibility. 

The standardized effect size for the credibility 
intervention on perceptions of evaluator credibility was 1.09 
((33.19 - 23.64)/( (9.49 + 8.00)/2)). The comparable effect 
sizes for high-level administrators and certified program 
evaluators were 1.48 and 1.00, respectively. These are very 
large effect sizes. 
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Second^ the results of the present study indicated that 
the issue- involvement intervention failed to directly affect 
perception of involvement^ as noted previously and also 
reported in Table X. This finding is noteworthy given recent 
research which has indicated that issue involvement is an 
important. motivational variable in changing how people 
cognitively respond to information (Petty & Cacioppo^ 1986; 
Greenwald, 1968) • 

This finding is particularly noteworthy given the multiple 
correlation '(R « .48) between scores on the involvement scale 
and the five evaluation recommendations. This relationship was 
greater than the association (R « .36) between the credibility 
scale and the recommendations. Of course^ it is not possible 
to determine conclusively whether involvement affects reaction 
to recommendations or preferences for certain recommendations 
yield greater involvement. However, these results do suggest 
the importance of understanding involvement phenomena in an 
evaluation context. 

More studies emphat^iiiing ecologically valid involvement 
manipulations are needed if we are to understand the subtle 
influences on involvement in a reality in which authority is 
shared and change is incremental. What are. not needed are more 
studies offering stark but unrealistic contrasts of involvement 
conditions. 

The subtlety of involvement dynamics was indicated in an 
indirect way by the cognitive response data. The target issue 
appeared to be of interest due to the extent of cognitive 
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responses. Subjects in the hi-^h credibility group;>^ wero less 
likely to counterargue than the subjects in the low credibility 
groups. Also, subjects tended to make fewer overall responses 
if they perceived the source to be credible. 

Third, the findings of the present study supported the 
hypothesis that attitude change is a function of the extent and 
v.irection of cognitive responses. For all groups, more of the 
thoughts elicited were counterarguments to career option plans. 
But counterarguments were even greater for the low credibility 
condition than for the high credibility condition. This 
finding is also partially explained by dissonance theory 
(Pestinger, 1957), which posits that inconsistency creates 
tension until the individual resolves the conflict. When a 
source is perceived to have little credibility, knowledgeable 
subjects with more strongly held prior opinions can be expected 
to generate counterarguments. However, if the source is 
perceived to be highly credible, fewer counterarguments can be 
expected. 

Overall, the results consistently indicate that certified 
program evaluators and high-level administrators are more 
similar than dissimilar in their perceptions of evaluation 
information. Both role groups were aware of source 
credibility. Evaluators that were perceived to be 
underqualif ied were not as trusted by the role groups, and 
distrust led to counterarguments, especially if the 
communication was negatively perceived. Counterarguments are 
particularly important in an evaluation context, bacause these 
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cognitive responses may arise in response to a written 
evaluation report or when the evaluator is not present to 
respond to c6un*:erar9araent. 

The results in the present study suggest that evi?.iuator 
credibility is perceived by report readers, and that these 
perceptions may even affect counterargumentat ion. PuLthermore, 
counterargument may lead to inaction or delay even when dealing 
with favorably received recommendations. Clearly, these 
results suggest that evaluators miist actively and 
systematically endeavor to present themselves to administrators 
as being credible aources of information, and evaluators must 
consider the potential effects of counterarguments when writing 
their reportis. 
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Table 1 

Cell and Margin Means on Credibility and Involvement 



Dependent Variable: Credibility 
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27.39 (n=33) 


21.58 (n<7) 
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33.56 (n=16) 
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Independent Variable: 
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Administrator 24.50 (n=4) 


26.14 (n=7) 


34.80 (n=5) 


38.89 (n=4) 


Administrator 


30.53 (n=20) 
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[9.26] 
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[9.221 


Evaluator 20.69 (h=13) 


24.98 (n=13) 
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31.14 (n=14) 
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Total 
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Dependent Variable: Involvement 
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Total 


22.17 (n=7l) 












[13.92] 



Note. -Standard deviations are presented in brackets. 



Figure 1 
Geographic Distribution 




Note. The first number represents total responses per area. 

The second number represents administrators per area. 
The third number represents evaluators per area. 
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